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Abstract 

We prove optimality principles for semicontinuous bounded viscosity solutions of Hamilton- 
Jacobi-Bellman equations. In particular we provide a representation formula for viscosity 
supersolutions as value functions of suitable obstacle control problems. This result is applied 
to extend the Lyapunov direct method for stability to controlled Ito stochastic differential 
equations. We define the appropriate concept of Lyapunov function to study the stochastic 
open loop stabilizability in probability and the local and global asymptotic stabilizability (or 
asymptotic controllability). Finally we illustrate the theory with some examples. 
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1 Introduction 

We consider a TV-dimensional stochastic differential equation 

dX t = f{X t )dt + a{X t )dW t 

where Wt is a standard M-dimensional Brownian motion. Since the 60th, it was developed a 
stochastic Lyapunov method for the analysis of the qualitative properties of the solutions of 
stochastic differential equations, in analogy to the deterministic Lyapunov method. The main 
contributions are due to Hasminskii (see the monograph |21j and references therein) and Kushncr 
(see the monograph J2S] and |27|). They introduced the notion of stability in probability and 
asymptotic stability in probability. This means that the probability the trajectory leaves a given 
neighborhood of the equilibrium is decreasing to as the initial data is approaching the equilib- 
rium. If, moreover, the trajectory is asymptotically approaching the equilibrium with probability 
decreasing to as the initial data is approaching the equilibrium the system is asymptotically 
stable in probability. Finally if for every initial data the trajectory is asymptotically approaching 
the equilibrium almost surely, the system is asymptotically stable in the large. The stochastic 
analog of deterministic Lyapunov functions V are twice differentiable continuous functions, which 
are positive definite and proper and satisfy the infinitesimal decrease condition 

-DV{x) ■ f{x) - trace [a{x)D 2 V{x)} > l{x), (1) 
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with I > for mere Lyapunov stability and I > for x 7^ for asymptotic stability, where 
a := <7<7 T /2. By the Dynkin formula, this differential inequality implies that the stochastic process 
V(X t ) 1 where X t is the solution of the stochastic differential equation starting from x, is a positive 
supermartingale. This fact translates, in the stochastic setting, the requirement on the Lyapunov 
function to decrease along the trajectories of the dynamical system. There is a large literature on 
this kind of stochastic stability: we refer to the cited monographs and to [20], see also references 
therein. We recall here also the work of Florchinger I2()j and Deng, Krstic, and Williams [T^j 
on feedback stabilization for controlled stochastic differential equations by the Lyapunov function 
method. 

In this paper we extend the Lyapunov method for stochastic differential equations essentially 
in two directions. First of all we consider controlled stochastic differential equations in M 

dX t = f(X t , a t )dt + a(X t ,a t )dW t , 

moreover we allow the Lyapunov functions to be merely lower semicontinuous. The nonexis- 
tence of smooth Lyapunov functions is well known in the deterministic case, see |2] for stable 
uncontrolled systems and the surveys E] for asymptotically stable controlled systems. Also 
in the stochastic case, the assumption of smoothness for Lyapunov functions is not necessary and 
would limit considerably the applicability of the theory and the possibility of getting a complete 
Lyapunov-characterization of the stabilizability in probability by means of a converse theorem. 
Kushner proved in |26| a characterization of asymptotic uniform stochastic stability (for uncon- 
trolled systems) by means of only continuous Lyapunov functions (here, however, the infinitesimal 
decrease condition is not given with a differential inequality but in terms of the weak generator of 
the process). For stability in probability, Hasminskii provided a C 2 Lyapunov function under the 
assumption of strict nondegeneracy of the diffusion: this result cannot be extended to possibly 
non-degenerate diffusions. Converse theorems in the controlled case will appear in the Ph.D. thesis 
by the author |1U| . In particular we prove that the existence of a local Lyapunov function is also 
necessary for the stability in probability. Hence we show that if the system (CSDE) is uniformly 
asymptotically stabilizable in probability then there exists a local strict Lyapunov function, which 
is continuous. 

We define then a Lyapunov function for the stability in probability as a lower semicontinuous, 
positive definite, proper function V, continuous at and satisfying in viscosity sense the differential 
Hamilton- Jacobi-Bellman inequality 

max{-DV(x) ■ f(x, a) - trace \a(x, a)D 2 V{x)] } > l(x), (2) 

and we call it strict Lyapunov function if Z > off 0. Our main results are the natural extensions 
to the controlled diffusions of the First and Second Lyapunov Theorems: 

the existence of a local Lyapunov function implies the ( open loop ) stabilizability in probability of 
( CSDE); a strict Lyapunov function implies the (open loop) asymptotic stabilizability in probability. 

This means that if there exists a Lyapunov function, then for every initial data we can find 
an admissible control which keeps the whole trajectory near the equilibrium with probability 
decreasing to zero as the starting point of the trajectory approaches the equilibrium. Moreover, 
the existence of a strict Lyapunov function implies also that, for every e > 0, there exists an 
admissible control driving the trajectory asymptotically to the equilibrium with probability greater 
than 1 — e. The same proof provides the global versions as well: if V satisfies in M N \ {0} then 
(CSDE) is also (open loop) Lagrange stabilizable, i.e. has the property of uniform boundedness 
of trajectories, and if V is strict then the system is (open loop) asymptotically stabilizable in the 
large. We also give sufficient conditions for the stability of viable (controlled invariant) sets more 
general than an equilibrium point. 

The main tool to provide such a result is a supcroptimality principle for lower semicontinuous 
bounded viscosity supersolutions V of the Hamilton- Jacobi-Bellman equation J2J • The new point 
is that this superoptimality principle holds as an equality and then gives a representation of such 
V as value functions of an appropriate obstacle control problem. A similar approach has been 



2 



exploited in the deterministic case by Barron and Jensen (see 0) for globally asymptotically stable 
systems affected by disturbances and by Soravia ([HH|> [HI]) for stable systems with competitive 
controls. 

Precisely we prove that 
every bounded LSC viscosity supersolution V of 0) can be represented as 



This representation formula is important on its own, since it refers to Hamilton- Jacobi-Bellman 
equations for which it is not expected uniqueness of solutions. In particular this formula permits 
to give a characterization of the minimal nonnegative LSC viscosity supersolution of the equation 



The representation formula (JSJ is obtained by introducing a suitable sequences of obstacle 
problems, solved in the viscosity sense by V. The conclusion comes from a uniqueness results for 
viscosity solutions of such problems and from an approximation procedure. The delicate point is 
the proof of a suboptimality principle for the min-max value function (on the right of equality 
To get this principle, we have to choose appropriately the controls we are allowing for our 
problem. Actually the class of controls on which we minimize a certain given functional has to 
satisfy two key properties in order to get a dynamic programming principle. They are the stability 
under concatenation and stability under measurable selection. For the definition of the classes of 
controls and for the compactness and measurable selection results we are going to use, we refer 
mainly to the article by Haussmann and Lepeltier [221 ( see a ^ so the article by El Karoui and others 
[Tfij and the book by Stroock and Varadhan [301 ch 12]). For related results on the existence of 
optimal controls for stochastic problems we refer to the article by Kushner |28j . 

There is a large literature on dynamic programming and superoptimality and suboptimality 
principles for viscosity solutions of second order Hamilton- Jacobi-Bcllman equations, starting from 
the papers by P.L. Lions [21] (see also the book 53)- We recall here the recent work by Soner 
and Touzi on dynamic programming for stochastic target problems ([22], (US])- We refer also to 
the paper by Swiech |41j on superoptimality and suboptimality principles for value functions of 
stochastic differential games (see also the paper by Fleming and Souganidis HZ|). In this paper 
we are extending to the stochastic case some results obtained by Soravia in [3S] (see also [3~§| 
and [3|). He provides, in the general context of differential games, a representation formula for 
supersolutions of first order Isaacs equations. This gives a superoptimality principle which holds 
as an equality and which refers to equations which in general have not unique solutions. This 
result is then applied to the Lyapunov characterization of the stabilizability to an equilibrium set 
of a deterministic dynamical system with competitive controls by means of Lyapunov functions 
which are only required to be continuous on the boundary of the equilibrium set. 

In the last section, we present a simple application of our Lyapunov method. We consider an 
asymptotically controllable deterministic system and we study under which conditions it remains 
stable if we add to it a stochastic perturbation. By the Lyapunov characterization of asymptotic 
controllability provided by Clarke, Ledyaev, Rifford, Stern (J2[) and Rifford (dU), we know 
that the unperturbed system admits a Lyapunov function, which is Lipschitz continuous and 
semiconcave except possibly at the origin. We study therefore under which perturbations this 
function remains a Lyapunov function also for the perturbed system. In particular we get a small 
intensity condition on the diffusion matrix a, depending on the semiconcavity constant of the 
Lyapunov function and on qualitative properties of the stable trajectories of the deterministic 
systems. 

We conclude with some additional references. We recall that there are other notions of stochas- 
tic stability. Kozin introduced the exponential almost sure stability of uncontrolled stochastic 
system. The stability in mean square and the p-stability were studied by means of Lyapunov 




(3) 



(J2J) as the value function 
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functions (we refer to the monograph |21jV In the controlled case, in previous papers Bardi and 
the author (see 0) characterized by means of appropriate Lyapunov functions the almost sure 
stabilizability of stochastic differential equations. This is a stronger notion of stochastic stabil- 
ity, never verified for nondegenerate processes. Indeed a system is almost surely stabilizable if 
it behaves as a deterministic stabilizable system and remains almost surely in a neighborhood of 
the equilibrium point. Turning to deterministic controlled systems, a complete Lyapunov char- 
acterization of the asymptotic stabilizability (called asymptotic controllability) has been proved 
by Sontag and Sussmann (see the articles |33|, [HE] and the review paper (HEl)- The infinitesimal 
decrease condition of the Lyapunov function along the trajectories of the system is expressed in 
terms of Dini directional derivatives, contingent directional derivatives and proximal subgradients. 
There is a large literature on the stabilization of deterministic controlled system by the Lyapunov 
function method: we refer to the monograph |2j, to the papers [S], see also the references 
therein. 

The paper is organized as follows. In Section 2 we introduce the stochastic control problems 
and recall the definitions and the basic properties of the controls we are using. Section 3 is devoted 
to the proof of the representation formula (J3J for bounded continuous viscosity solutions of the 
differential inequality (J2J) and then to the extension to lower semicontinuous functions. Section 4 
contains the definitions of stabilizability in probability, asymptotic stabilizability and Lyapunov 
functions; in Section 5 we apply the results obtained in Section 3 to prove local and global versions 
of the Lyapunov theorems. We show that, given a control Lyapunov function or a strict control 
Lyapunov function, the system (CSDE) is respectively stabilizable or asymptotically stabilizable 
in probability. In Section 6 we introduce the notion of controlled attractor and we discuss the 
generalization of the direct Lyapunov method to the case of stabilization of viable sets. Finally in 
Section 7 we present some examples illustrating the theory. 



2 Stochastic control setting 

In this section we introduce the stochastic control problem and recall the definitions and the basic 
properties of the controls we are using. 

We consider a controlled Ito stochastic differential equation: 

(CSDE) { dX t = f(X t ,a t )dt + a(X t ,a t )dB t , t > 0, 

We assume that at takes values in a given compact set A C Ft , /, a are continuous functions 
defined in Ft N x A, taking values, respectively, in Ft N and in the space of N x M matrices, and 
satisfying for all x, y € Ft N and all a 6 A 



We define 



and assume 



\f(x, a) - f(y, a)\ + \\a{x, a) - a(y, a)\\ < C\x - y\. (4) 
1 



a(x.a) :— —a(x, a)a(x, a) 



T 



{(a(x, a), f(x, a)) : a £ A} is convex for all x € Ft . (5) 

We recall here the definition of admissible controls that we are allowing for our control problems. 
For precise definitions we refer to [221 an d |E3 ( see also references therein). Actually in these 
articles the problem is formulated in terms of solutions of the martingale problem, but it is also 
showed that there is an equivalent formulation in terms of solutions of (CSDE). 

We are relaxing the control problem by using weak controls, that is, admitting all the weak 
solutions of (CSDE). We have not assigned a priori a probability space (fi, T , Tt, P) with its 
filtration. So when we introduce a control we mean that we are at the same time choosing 
also a probability space and a standard Brownian motion B t on this space. Actually under the 
hypothesis Q it can be shown that the space of strong controls is not empty and that, under 
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suitable assumptions on the cost functional (which are essentially the lower semicontinuity with 
respect to the x variable), the strong problem and the weak problem have the same value ( [161 
Theorem 4.11]). 

Definition 1 (Strict controls, Definition 2.2 [22). For every initial data x £ M N , a strict 
control is a progressively measurable A-valued process {pct)t>o such that there exists a M N valued, 
right continuous, almost surely continuous, progressively measurable solution X" to (CSDE) (see 
also [16, Definition 1.4]) ■ We denote with A x the set of strict controls for x £ M N . 

The class of strict controls can be embedded, as in the deterministic case, in a larger class of 
admissible controls. We denote by M(A) the set of probability measures on A endowed with the 
topology of weak convergence. We note that it is a separable metric space. 

Definition 2 (Relaxed controls, Definition 3.2 [22]). For every initial data x S M N , a 
relaxed control is a progressively measurable M(A)-valued process (nt)t>o such that there exists 
a M valued, right continuous, almost surely continuous, progressively measurable solution 
to (CSDE) (see also [Iff, Definition 2.4])- We denote with M x the set of relaxed controls for 
xeM N . 

We choose now a canonical probability space for our control problem. By means of this 
canonical space we can give a formulation of the optimization problem in a convex compact 
setting. The most natural canonical space for the strict control problem seems to be the space 
of the trajectories X. of (CSDE). ft is the space of continuous functions C from [0, +oo) to M N 
with its natural filtration. In order to give a control on this space, it is sufficient to specify the 
probability measure on C (which is the law of the process X, ) and the progressively measurable 
function a. Rather than working with this canonical space we consider the space of trajectories 
(X.,fj,,) for fi relaxed control. Let V the space of measurable functions from [0, +oo) to M(A) 
with its canonical filtration. We denote with M(V) the set of probability measures on V endowed 
with the stable topology (this is a topology introduced by Jacod and Menin, for precise definition 
we refer [221 section 3.10] and references therein). The canonical space for the relaxed control 
problem will be the product space C x V with the product filtration. We call canonic relaxed 
control or control rule a relaxed control defined in this canonical space (see [221 Definition 3.12] 
and also ^3 Definition 3.2]). In order to identify a canonic relaxed control, it will be sufficient to 
specify the probability measure on the space C x V: the canonic relaxed controls can be considered 
as measures on the canonical space. 

In the following we will consider a cost functional 

V{X t ) + f l(X s )ds 



J(x, a) = sup 
t>o 

where I is a continuous, nonnegative function and V is a LSC, nonnegative function. The functional 
j(x,t, a) = ~Ei x \V{Xt) + J Q l(X s )ds] satisfies, for every x and t, the lower continuity assumptions 
required in [22| on the cost functional. Then, since the supremum of LSC maps is LSC, we get 
that also the functional J{x, a) satisfies the same lower semicontinuity assumptions. We list here 
the results obtained in [22] that we are going to use. The crucial assumption for all of them, 
besides the right choice of the class of admissible controls and the lower semicontinuity of the cost 
functional, is the convexity assumption JSJ. 

The class of control rules is the class on which it is possible to formulate a dynamic programming 
principle and to show the existence of an optimal control. The key result is Proposition 5.2 in 

m 

for every initial data x, the class of optimal control rules admissible for x is convex and compact. 
We have the following theorem stating the existence of an optimal control. 

Theorem 3 (Theorem 4.7 and Corollary 4.8 [22]). Under the convexity assumption and 
the other assumptions listed above, for every initial data x £ M N there exists an optimal control 
rule for the control problem 

inf J(x, a). 
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Moreover the infimum of the cost functional computed on the class of control rules coincides with 
the infimum of the cost functional computed on the class of strict controls: 

inf J(x,a) = inf J{x,a). (6) 

In particular the optimal control can be chosen strict. 

The two crucial properties on the control space to get a dynamic programming principle are 
the stability under measurable selection and the stability under concatenation (see [231 ) ■ They are 
satisfied by the class of control rules. We consider a measurable set valued map from M N to the 
space of probability measures on the canonical space C x V, with convex compact values. Then, 
by a standard measurable selection theorem (see ^Hl Theorem 5.3]) this map has a measurable 
selector. In 22 Lemma 5.5] (see also 0U1 cn 12] and ^3 Theorem 6.3, 6.4]) it is proved that 
this measurable selector is an admissible control rule. Moreover, in |22l Lemma 5.8] (see also ^3 
Theorem 6.2]) it is shown that if we take an admissible control and then at some later stopping 
time we switch to an e-optimal control from then on, the concatenated object is still admissible. 

Finally we observe that all these results remain valid if we consider instead of the trajectories 
of (CSDE) in M N , the trajectories of this system stopped at the exit time from a given open set 
(see |22]L 



3 Superoptimality principles 

In this section we prove a representation formula for bounded LSC viscosity supersolutions of 
Hamilton- Jacobi-Bellman equations. We start proving the result for continuous functions and 
then, by a standard approximation procedure, we extend it to LSC bounded functions. The 
representation formula is given first in the whole space M N and then also in a localized version. We 
are adapting to the second order case the proof of optimality principles for viscosity supersolutions 
of first order Hamilton- Jacobi equations given by Soravia in |38| and |39j . This requires the use 
of stochastic control instead of deterministic control. 

We consider the following Hamilton-Jacobi-Bellman equation 

max{-/(i,a) • DV(x) - trace[a(x,a)D 2 V(x)]} - l{x) = 0, (7) 
where I : JR N — ► IR is a nonnegative bounded continuous function. 

Theorem 4 (Representation formula for viscosity supersolutions). Consider a bounded 
LSC function V : M N — * M. If V is a viscosity supersolution of the Hamilton-Jacobi-Bellman 
equation in M N , then it can be represented as 

rt 



v(xn + / i(x?)ds 



V(x) — inf sup E 2 
a t>o 

where the infimum is taken over all strict admissible controls. 



(8) 



Proof. Without loss of generality, we can reduce to the case V > by an appropriate translation. 
Since V is LSC, bounded and nonnegative, we can consider an increasing sequence of continuous, 
nonnegative, bounded functions Vk such that 



V(x) — sup Vk(x) for every x S M N . 

If V is continuous, we choose Vk — V for every k. 

Now for every fc > 0, we introduce the following obstacle problem in Ml with unknown W 
and obstacle 14: 

mm{\W(x) + max \—f(x, a)DW(x) - trace a(x, a)D 2 W(x)] - l(x), 

W(x)-V k (x)}=0. (9) 
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Obviously V is a bounded LSC viscosity supersolution of the problem © for every A > and 
every k > 0. 

For A > fixed, define 



inf sup E K 

a t>o 



e- M V k (X?) + / l(x:)e- As ds 



The plan of the proof is the following. First of all we show that the value function L\k is 
a bounded discontinuous viscosity subsolution of the obstacle problem 0: this means that the 
upper scmicontinuous envelope L x k is a viscosity subsolution of . Then, by the comparison 
principle for bounded discontinuous viscosity solutions of Isaacs equations we get that, for every 
A > 0, L\ t k{%) < V{ x )- From this, we can conclude, sending A to and k to +oo, that V satisfies 
the superoptimality principle ijSJ). 

By the definition and the boundness of Vk, we get that L\,k is bounded. We want to prove that 
its upper semicontinuous envelope L* x k is a viscosity subsolution of the obstacle problem . To 
get this result it is sufficient to check that L* x k is a viscosity subsolution of the Hamilton-Jacobi- 
Bellman equation 



XLix) + max {—fix, a) ■ DL(x) — trace\a(x, a)D 2 L{x)]\ — Ux) = 

aeA 



(10) 



at the points x where L\ k (x) > Vk(x). This result can be obtained by standard methods in the 
theory of viscosity solutions if we prove a local suboptimality principle for L* x k on such points x 
(see U3] and 

First of all wc need the following technical lemma whose proof we postpone to the end. 

Lemma 5. If k {x) > Vk(x) then there exists a sequence x n — > x with L\^(x n ) — ► L* x ,{x) and 
L\,k{x n ) > Vk(X n ) for which there exists e > such that 



L\.k{x n ) < inf 



-Xt T * 



L\A X n{t)) 



l{X«{s))e^ s ds 



(11) 



for t < e and \x n — x\ < e. 



This is a local suboptimality principle. This inequality, by a standard argument in the theory 
of viscosity solution (for the detailed argument see for example 0, see also JZ]), implies that L\ k 
is a viscosity subsolution of equation at the points x such that L\ k {x) > Vk(x). From this we 
deduce that L* x k is a viscosity subsolution of ©. 

By the comparison principle obtained for Isaacs operators and bounded discontinuous viscosity 
solutions by Ishii in j^J Theorem 7.3], we get that V(x) > L* x k {x) for every A > and k > 0. In 
particular, for T > and k fixed, we get 



V(x) > lim L\ k(x) > lim inf sup E 2 

A^O ' A^O a te[ o, T ] 



-A/ 



l{X^)e- Xs ds 



> 



> lim e AT inf sup E a 

Q te[o,T] 



VkiXf) 



Therefore for every T > and k > 



l(X?)ds 



V(x) > inf sup E x 
a te[o,T] 

Now we want to pass to the limit for k — > - 
admissible control OLk such that 



1 



V k (X?)+ / l(X°)ds 



-oo. For x fixed and every k > we consider an 



V(x) + - > sup E a 
fc te[o,T] 



Jo 



(12) 



7 



By the definitions recalled in Section 2 we can associate to each couple (X ak ,a k ) a control rule 
P k . By the compactness of the space of control rules, we can extract a subsequence of control 
rules, which we continue to denote with P^, to some control rule P in the stable topology. This 
control rule is the measure of a trajectory of (CSDE) (X, , n) driven by a relaxed control fi. Since 
the convergence in the stable topology implies in particular the weak convergence of the measures 
P k to the measure P, we get immediately that for every t £ [0, T] 



lim E x / l(X* k )ds = E x I l(X s )ds 

where the expected value on the left and right hand side is computed, respectively, using the 
measures Pk and P. 

Recalling now that V = sup fe V k where V k are continuous functions, it is easy to show that V 
can be obtained as 



V(x) 



lim inf V k (y) := sup inf i V k (y) 

k — >+oo, y — >x 



\x — y\ < 6 k > 



1 



Moreover, since the convergence in the stable topology implies also the convergence in probability 
of Xf h to X t , we get that, for t £ [0, T] fixed, we can extract a subsequence X" k which converges 
to X t almost surely with respect to the measure P. Then along this subsequence 

lim inf V k (X? k ) > lim inf V k (y)>V(X t ) P almost surely. 

k — >+oo k — >+oo,y — >Xt 

By the Fatou lemma and the definition of stable convergence we deduce that, for each t £ [0,T], 
along a subsequence 

lim inf EV k (X" k ) > EV(X t ), 

k — >+oo 

where the expected value is computed respectively using the measures P k and P. 
To summarize, for every t £ [0, T] we get from <I12[) : 



V(x)> lim E x V k (X? k )+ / l{Xf k )ds 

fe^+oo L JO 

So for every T > there exists a control rule for which 

V(X t ) 4 

Now, by the statement © in Theorem |31 we obtain 



> E T 



V(X t ) 



l(X s )ds 



V(x) > sup E 2 
te[o.T] 



l(X s )ds 



V{x) > inf sup E x 
a te[o,T] 



V(X?) + / l{X«)ds 



(13) 



Now it remains only to let T — > +oo. For e > 0, consider an e optimal control a for 1)13(1: in 
particular it gives V r (x) + § > ~E X \ V(X%) + f*l(X?)ds\. Conside ring X% as starting point of the 



trajectory we obtain by (|13fl 



V(X$) > inf sup E x ° 
P te[o,T] 1 



V(XP) + / l{Xfi)d8 



a.s. 



Let j3 be an ^ optimal control rule for V(X^) (we can choose a measurable selection of the opti- 
mal controls rules for V(X^)(u>)), moreover the control rule obtained concatenating this selected 
control and a is still an admissible control rule. We can then proceed recursively and conclude by 
induction that we can construct an admissible control rule P such that 



V(x) + e> sup E a 
t>o 



V(X t ) 



l(X s )ds 
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By the statement (0 in Theorem |3 and recalling that e is arbitrary, we obtain 



V(x) > inf sup E x 
a t>o 



V(X t ) + / l(X s )ds 



> V(x) 



which is the desired formula. 

We give here the proof of the technical Lemma 



(14) 
□ 



Proof, of Lemma If the statement were not true, for every sequence x n — > x such that 
L\ t k{x n ) — > L\ k (x) and L\. k (x n ) > Vk(x n ) for every n we could find t n < ^ such that for 

\%n ~ X\ < ^ 



L\,k{x n ) > inf E x „ 



As. 



By definition of L\,k, for every e n > and every control a, there exists T(e„, a) such that 



- XT ^V k (X%(T(s n , a))) + / ;(X«( S ))e- As d S 



T(e„,a) 



< E 



By the inequality (|15|) . we can choose a sequence e n — > and controls /3 n for which 



- At "Lt fe (X r f"(t„))+ / l(X^(s))e- Xs d S 



Therefore for every control a we obtain from the inequalities Ijl6|) and l|17|l 



(15) 



(16) 



(17) 



E , 



e-*~Ll k (Xfr(t n ))+ I l(X^(s))e-^ds 

■T( Sn 



< E, 



-\T(e„,a) v ,{Ya 



V k {X%(T(s n ,a))) + 



+ £n < Lx,k{x n ) 



ZpC( S ))e- As ds 



< 



We claim now that for every n there exists a such that T(e ni a) < t n . Assume by contradiction 
that there exists N such that, for every a admissible, T(sn, a) > tjy, in particular T(sn, a at) > t/v 
for every control a at which for t < tjv coincides with /3/y: by the previous inequality we get 



^x N L\ k {X N N (t N )) + e N < E XN 



-A(T(eiv,Qiv)— *n)t/. f V a « 



V k (X%»(T(e N ,a N )))+ 



+ 



l(X^ N (s))e- x ^ tN) ds 



We choose now an admissible control rule which is £at/2 optimal for L\ u{Xjf (tjv)) (we operate 
a measurable selection between the £at/2 optimal control rules for L* x k (X^ N (tN))(oj)) and the 
concatenated control with (3n is still an admissible control rule Pjy which is the measure associated 
to the couple (Xn(-), Hn)- Therefore we get 



sup Ejy 

t>t N 



< E 



JV 



e- M V k (X N (t))+ / l(X N ( S ))e- Xs ds 



T(ejv,Aijv) 



e N /2 < 



- XT ^'^V k (X N (T(e N ,P N )))+ / l(X N (s))e- Xs ds 
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where the expected value is computed with respect to the measure P/v, and the inequality we 
obtain gives a contradiction. Therefore there exists for every n an admissible control rule P n such 
that T(e„, fx n ) < t n : choosing a = /i„ in the inequality we get 



L\.k{x n ) — £n < Ea 



-AT(s„,ft„) 



y fe (X„(T(e n , Mn ))) 



l(X n {s))e 



-As 



For every n we call A n — {lu\ X n (T(s n , fj, n )) € B{x n , (tt^)} an d B n = Q, \ A n : since for every 
n the trajectory is a Markov process and the drift and the diffusion of this control problem are 
equi-Lipschitz and equi-bounded in the compacts with respect to the control it is possible to show 
(we refer to [H pp 284,285] for the proof) that P(B n ) < KO(^f where O(t^) 3 is uniform 
with respect to the initial data x n and to the control [i n . Therefore we get 



T I \ n 

L\,k(Xn) ^" < 



V k {X n (T(e n ,fi n )))- 

T(£„, M „) 



V k (X n (T(e n ,fM n ))) 



l(X n (s))e- Xs ds 
l(X n (s))e- Xs ds 



< 



sup V k {y)+ sup l(y)t„ 



from which, since V k is continuous and t n < l/n, letting n 

Ll k (x)<V k (x) 

in contradiction with our assumption. 



-00, we deduce 



□ 



Remark. The previous result can be proved in more general situations: consider a bounded, 
nonnegative, LSC viscosity supersolution V : M N — > M of 

max {-fix, a) ■ DVix) - trace\aix,a)D 2 V ix)] \ + k(x)Vix) > l(x) 

a£A L ' 

where k : M N — > M is a Lipschitz continuous nonnegative function. The proof of Theorem 0] 
applies directly and we obtain the representation formula 



V(X?)e-ti k W dB + / l(X^)e--fo k ( x ^ du ds 



V(x) — inf supE 2 

Q t>0 

We can prove also a localized version of the Theorem 

Corollary 6. Consider an open set O C M N . For every S > 0, consider the set Os ■= {x € 
O I dix, dO) > 6} and denote with Tf the stopping time at which the sample function of the 
process Xf reaches the boundary dOs'- we denote with rfit) the minimum between rf and t. 
Assume that V : O — > 1R is a bounded nonnegative function. IfV is a LSC viscosity supersolution 
of the Hamilton-Jacobi-Bellman equation in O , then it can be represented, for every S, x G Os, 
as 



V(x) = inf supEj 



t>o 



Proof. We fix 6 > and a smooth cut off function < ^ < 1 such that = for x E M N \ O 
and £(x) = 1 for x £ Os- We consider the stochastic controlled differential equation in M N : 



(CSDE)' 



dX t = fiX t ,a t )e(X t )dt + tr(Xt,at)Z(Xt)dB t , t > 0, 
X Q = x. 
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Observe that for x E 0$, the solution (X') a to (CSDE)' coincides a.s. with the solution X a to 
(CSDE) up to time t£ . We define the process X" a ^ obtained by stopping the process (X')f at 
the instant it reaches the boundary of Og: it has a Ito stochastic differential and it is still a strong 
Markov process with continuous trajectories (see for example |21l Lemma 3.3.1] and references 
therein). 

We extend V outside O as a bounded nonnegative LSC function that we continue to denote 
V. So it is immediate to show that V is a viscosity supersolution in M N of the equation: 

max{-f(x,a)^(x) • DV{x) - trace[a{x, a)f(x)D 2 V(x)]} - l(x)£ 2 (x) = 0. (18) 
We can apply Theorem 0] to V. Indeed it is sufficient to define 



L\,k(x)={ 



in O s 



M a su Pt > E x [e-*?®V k (X? r(A ) + J* l(X^ a) )^(X° ?w ) e - Xr *^da _ 
V k {x) ' in m N \o 5 



We can repeat the proof in Theorem 0] (all the results in [22 hold also for the stopped process 
Y a ) and we get that L* x , is a viscosity supersolution of the obstacle problem ijjjj in M N . So again 
repeating the same arguments of Theorem 0] we get that V satisfies the following representation 
formula for x E Os- 



V(x) = inf supE 2 



n;v:' Vli] i - / \xf)ds 







□ 



Remark (Minimal nonnegative solution). These representation formulas for viscosity solu- 
tions are interesting on their own, as we have pointed out in the introduction: indeed they apply 
to Hamilton-Jacobi-Bellman equations for which there are no comparison principles and then no 
uniqueness of solutions. 

We consider the following Hamilton-Jacobi-Bellman equation in M N 

max {—f(x, a) ■ DV{x) - trace[a(x, a)D 2 V(x)}} = l(x) 

with / > 0: since the constant function U = is always a subsolution, it is interesting to charac- 
terize the minimal nonnegative supersolution. 

From a control point of view, the natural solution seems to be the value function of the infinite 
horizon control problem with running cost / 

r+oo 

V 00 (x)=M-E x / l(X«)ds. 
Jo 

If Vqc is well defined and bounded, then it is possible to show that it is LSC, by an argument based 
on the properties of the class of admissible relaxed controls. Moreover, by standard methods in the 
theory of viscosity solutions (see ^hES), ^ ^ s P oss ible to show that Voo is a viscosity supersolution 
of the previous Hamilton-Jacobi-Bellman equation. 

In this case, an easy application of the previous theorems gives that every bounded, nonnega- 
tive, viscosity supersolution V of the Hamilton-Jacobi-Bellman equation in M N satisfies 

V(x) > V^x) 

therefore is the minimal nonnegative viscosity supersolution of the equation. 

Remark (Representation formula for viscosity subsolutions). The counterpart of Theorem 
0] for viscosity subsolutions is straightforward from classical suboptimality principles: let U : 
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M N — ► 1R be an upper semicontinuous bounded viscosity subsolution of the Hamilton- Jacobi- 
Bellman equation 

max {-f(x, a) ■ DU{x) ~ trace[a(x, a)D 2 U{x)]} < l(x), 
then the function U can be represented as: 



U(x) = inf inf E a 

a t>0 



U{X?) + / l(X?)ds 



4 Stability in probability and Lyapunov functions 

We begin this section with the notion of both Lyapunov and asymptotic stability in probability. 
They were introduced by Hasminskii and Kushner (see [21] and [23]) in the case of uncontrolled 
stochastic differential equations. We present their natural extension to the case of controlled 
diffusions. 

Definition 7 (stabilizability in probability). The controlled system (CSDE) is (open loop) 
stabilizable in probability at the origin if for all e, k > there exists S > such that for every 
\x\ < 6 there exists a control a. G A x such that the corresponding trajectory X, verifies 



P x sup|X t | > k )<e. 
\t>o J 

This is equivalent to assume that for every positive k 

lim inf Pz (sup |X t Q | > k ) = 0. 
a \t>o ) 

The system is (open-loop) Lagrange stabilizable in probability, or it has the property of uni- 
form boundedness of trajectories, if for each e > 0, R > there is S > such that, for any initial 
point x with \x\ < R, 

miP x ( sup\X?\ > S ) < e. 
a \t>o J 

This is equivalent to assume that for every R > 



lim sup inf P., sup \X?\ > S) = 0. 

S^+oo | X |< R a \ t > J 

Remark. The stabilizability in probability implies that the origin is a controlled equilibrium of 
(CSDE), i.e., 

3a e A : f(0,a) = 0, a(0,a) = 0. 

In fact, the definition gives for any e > 0, for k > fixed, an admissible control such that the 
corresponding trajectory starting at the origin satisfies P(sup t>0 \X t \ > k) < e so 

"+00 



E ^ °° l(\X t \)e- xt dt < ~ 



for any A > and any real function / such that < l(r) < 1 for any r and l(r) = for r < k. 
Then inf Q e _4 Eo / + °° l{\X t \)e~ xt dt = 0. Theorem implies that the inf is attained: therefore for 
any k > there is a minimizing control which produces a trajectory satisfying a.s. \X t \ < k for 
all t > 0. So inf Q e ^ g Eo J Q °° \X t \e~ xt dt — for any A > 0. Again Theorem [3] implies that the 
inf is attained, and the minimizing control produces a trajectory satisfying a.s. \X t \ = for all 
t > 0. The conclusion follows from standard properties of stochastic differential equations. 
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Regarding the Lagrange stabilizability, we observe that, using standard properties of diffusions 
under the regularity assumptions it is possible to prove (see ^31^) that for every fixed T > 
and R > 



lim sup inf sup \X?\ > S) = 0. 

S^+oo \ X \< R a \0<t<T J 

Nevertheless the Lagrange stabilizability is a stronger condition since it requires that 



lim sup inf supP^ sup \X"\ > S I =0. 

s ^+°° \x\<R a T>Q \0<t<T J 

The controlled diffusion is said to be asymptotically stabilizable in probability if the equilibrium 
point is not only stabilizable but also an attracting point for the system, locally around the 
equilibrium point. 

Definition 8 (asymptotic stabilizability in probability). The controlled system is locally 
asymptotically stabilizable in probability at the origin if for all e,k > there exists S > such 
that for every \x\ < S there exists a control a, G A x such that the corresponding trajectory X 
verifies 

P x ( sup|X t | > k ) < e and P x (limsup \~X t \ > 0] < e. 
This is equivalent to assume that for all k > 

lim inf P, | sup ATI .•/••) -fP,. Mip |. V;' .-(») = 0. 

x—>0 a 



sup \X?\ >k) + P X [ limsup > 

t>Q J \ t^+oo 



There is a global version of the previous stability notion: 

Definition 9 (asymptotic stabilizability in the large). The controlled system is asymp- 
totic stabilizable in the large at the origin if it is Lyapunov stabilizable in probability around the 
equilibrium and for every x £ M N 



mfP x limsup|X t a | > = 0. 

This means that for every e > and for every initial data x we can choose an admissible control 
in A x which drives the trajectory to the equilibrium with probability greater than 1 — e. 

Next we give the appropriate definition of control Lyapunov functions for the study of the 
stochastic stabilities defined above. 

Definition 10 (Lyapunov function). Let O C M N be a bounded open set containing the origin. 
A function V : O — > Mi is a local Lyapunov function for (CSDE) if it satisfies the following 
conditions: 

(i) it is lower semicontinuous and continuous at the origin; 

(ii) it is positive definite, i.e., V{0) — and V{x) > for all x T 0; 
(Hi) it is bounded; 

(iv) it is a viscosity supersolution of the equation 

max{-W(;r) ■ f(x, a) - trace [a(x,a)D 2 V{x)]} =0 in O. (19) 

We introduce the notion of strict Lyapunov function both in the local and global setting. 

Definition 11 (local strict Lyapunov function). Let O C JR^ be a bounded open set con- 
taining the origin. A function V : O — > St is a local strict Lyapunov function for (CSDE) if it 
satisfies the conditions (i),(ii), (Hi) in the previous definition and 
(iv) it is a viscosity supersolution of the equation 

max {-DV(x) ■ f{x, a)- trace [a{x, a)D 2 V{xj\ } =l{x) inO, (20) 
where I : O — * M is a positive definite, bounded and uniformly continuous function. 
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Definition 12 (global strict Lyapunov function). Let O C M N be an open set containing 
the origin. A function V : O — > 1R is a global strict Lyapunov function for (CSDE) if it satisfies 
the following conditions: 

(i) it is lower semicontinuous and continuous at the origin; 

(ii) it is positive definite, i.e., V(0) — and V(x) > for all x ^ 0; 

(Hi) it is proper, i.e., \\m. x ^QQ V(x) = +oo, or, equivalently, its level sets {x\V(x) < /i} are 
bounded; 

(iv) it is a viscosity supersolution of the equation 

max {-DV(x) ■ f(x, a)- trace [a{x, a)D 2 V{x)] } =l{x) inO, (21) 

where I : O — > M is a positive definite uniformly continuous function. 



5 Direct Lyapunov theorems 

In this section we develop a direct Lyapunov method for the study of stabilizability in probability of 
controlled diffusions both in the local and global setting. For the uncontrolled case, the extension of 
the Lyapunov second method to the case of stochastic systems is due to Hasminskii and Kushner 
independently (see the monographs |U, El i see a l so the references therein for earlier related 
results) . 

The main tool of the proof of the Lyapunov theorems is the representation formula for viscosity 
solutions obtained in Section 

Theorem 13 (Stabilizability in probability). Assume conditions ^p, 0) and the existence 
of a local Lyapunov function V in the open set O. Then: 

(i) the system is stabilizable in probability, 

(ii) if in addition the Lyapunov function is global, then the system is also Lagrange stabilizable in 
probability. 

Proof. We start proving (i). We fix k > such that B). C O. We fix e > and define w = 
e m\n\ y \> k V(y). We denote with t£{x) the first exit time of the trajectory X" from the open 
ball Bk centered at the origin with radius k. By the continuity at the origin of V we can find 
9 > such that if \x\ < 9 then V(x) < The superoptimality principle in Corollary gives, for 
\x\<9Ak, 

ry/2 > V(x) - inf supE x V(X tA « (x) ). 

a t>o k 

We choose now an ^ optimal control a € A x for the previous control problem, we denote by X t 
the corresponding trajectory, stopped at the exit time from O, and we get for every t > 

n > E x V(X tAr7 ) > f _ V(X Tf ) > P( sup \X S \ > k) mm V(y). 

J{sup < s < t \X 3 \>k} 0<s<t \y\>k 

As t — > +oo, we obtain the following bound on the probability that the trajectory X t leaves the 
ball B k 

P(sup \X t \ >k)< ^— — = e. 

t>o min|j,|> fe V(y) 

This proves the stabilizability in probability. 

We pass now to (ii). Repeating the argument above we get that for every k > 

infP(sup|A7| > k) < . . 
a t>o mm| y |> fc V(y) 

This implies the Lagrange stabilizability: indeed given R > and e > 0, we choose k such that 

ma,x M < R V{y) 



< e. 

\ y \>kV{y) 



□ 
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In the case the system admits a strict Lyapunov function we prove that there exists a control 
which not only stabilizes the diffusion in probability but also drives it asymptotically to the 
equilibrium. We obtain the result using standard martingale inequalities; in the uncontrolled case, 
a similar proof of asymptotic stability has been given in ( see a ls° ED])- 

Theorem 14 (Asymptotic stabilizability). Assume conditions p)). ^jj and the existence of a 
local strict Lyapunov function in an open set O. Then 

(i) the system (CSDE) is locally asymptotically stabilizable in probability; 

(ii) if the strict Lyapunov function is global, then the system (CSDE) is asymptotically stabilizable 
in the large. 

Proof. We start proving (i). For every k > 0, such that C O, we get, by Corollary H3 that the 
function V satisfies, for x £ Bk, the following superoptimality principle 



V(x) — inf supE a 



(22) 



where the trajectories are stopped at the exit time from B^. By Theorem[2]there exists an optimal 
control a £ A x for this value problem. Wc indicate with X . the corresponding trajectory and with 
t the exit time from the open ball Repeating the proof of TheorcmllHIwe get the stabilizability 
in probability: 

P x (t < +oo) = P x (sup \X t \ >k)< . V( - X \,, r . 

i>o mm\ y \> k V{y) 

We denote by B(x) = {lo \ \Xt{w)\ < k Vt > 0}. By the previous estimate P x (B(x)) = 
P x (r = +oo) > 1 -V(x)/(mm lyl > k V(y)). 

We claim that l(X t (uj)) — > as t — > +oo for almost all u> £ B(x), from this, using the positive 
definiteness of the function I, we can deduce that 

limsuplXtl >o) < . V{X \ /( 
t^+oo / nmi| v |>fc V{y) 

which gives, by the continuity at the origin of the function V, the asymptotic stabilizability in 
probability. 

We assume by contradiction that the claim is not true: then there exists e > 0, a subset 
fi e C B(x) with P(f2 e ) > 0, and for every u> £ fl e a sequence t n {u) — + +oo such that l{X tn (u>)) > e. 
We define 

F{k) :— max |/(x, = max || cr(a?, ck) || . 

\x\<k,a£A \ X \<k,aeA 

We indicate with r(s) the minimum between r and s and compute, for t > fixed 

e{ sup \X- (s) ~X-(t)\ 2 \ = 

{t<s<t+h ) 
r r(s) r r(s) 



r r ( s ) r T ( s ) 

= E<^ sup 1/ f(X u ,a u )du+ a(X u ,a u )dB u \ 2 \ < 

[t<s<t+h Jr(t) Jr(t) J 

!r~( s ) ~) f r~( s ) ~) 

sup 1/ f(X u ,a u )du\ 2 \ +2E{ sup \ a(X u ,a u )dB u \ 2 } < 

t<s<t+h Jr{t) J \t<s<t+h Jr(t) J 

< 2F 2 {k)h 2 + 2e{ sup | f ( a(X u ,a u )dB u \ 2 \ . 

I t<s<t+h Jr(t) j 

By Theorem 3.4 in (15) (the process | o~(X u ,a u )dB u \ is a positive semimartingale) we get 
e( sup \X- {s) -X- (t) \ 2 \ < 2F 2 (k)h 2 + 8 sup e{| f a(X u ,a u )dB u \ 2 \ < 

{t<s<t+h J t<s<t+h I Jf(t) I 
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< 2F 2 (k)h 2 + 8 sup e{ / ( ' \o{X u ,a u )\ 2 du\ < 2F 2 (k)h 2 + 8T, 2 (k)h. 

t<s<t+h Jr(t) 



r(t) 

Then, Chebyshev inequality gives, for every t > fixed 



sup |X T(S) - X T(t) \>r) < U. L_d_. (23) 

4<s<i+/l / r 

Since £ is continuous, we can fix S such that \l(x) — l(y)\ < § if \x — y\ < S and |ac|, \y\ < k: we 
compute 

P ( sup \l(X ns) ) - l(X n t))\ < |) > P ( sup |X T(S) - X T(t) | < s) > 

\t<s<t+h *J \t<s<t+h / 

2F 2 {k)h 2 + 8Y, 2 (k)h 
- 1 S 2 ' 

-t-8E 2 (fc)fe ^ -p 

<5 



We choose h such that < 2F ( fc ) fe + 8S ( fc ) fe < P 2 ,(J7 e ) — r for some r > so that for every t > 



sup \l(X T(s) ) - l(X T(t) )\ < £ - ) > 1 + r - P X (0 E ). (24) 

\t<s<t+h 1 

From I122II . letting t — > +oo, we get 

Z(Z 5 )ds> / / 
* b(x) Jo Jn c Jo 



v{x) > / i(x s )d s > / i(x s )ds > / V / z(x s )ds > 

J b{x) Jo Jn c Jo Jn e n Jt n 

> [ V/i inf l(X t (u>)) > f inf Z(X t (u;)) > 

V „ [t»(w),t„(0»)+fc] „ V [t„M,t„(a>)+/ l ] 



r 



>/>-]Tp sup |TO-z(x 4 j|<- no £ >fc^- 

where the last inequalities are obtained using the strong Markov property of the process X TK r\. 
This gives a contradiction: then, for every e > 0, P(r2 £ ) = 0. We have proved that l(X t ) — > as 
t — ► +oo for almost all u> G B(x), now the positive definitcncss of I implies that 

pj lim \X t \=o\>P x (B x ) >1- 1 ,J '' 



11111 KM. ^ f Lt. \ IX. y . t./ \ ■ 

*^+°° J mm |y|>fcMZ/) 

We prove now the statement (ii). If O coincides with the whole space, arguing as above, we 
get that for every k > and x € there exists a strict control a k such that the corresponding 
trajectory X k verifies 

P x ( limsupZ(X t fe ) > ) < V{x)/ min V{y). 
Using the properness of the function V, by letting fc — > +oo, we get that for every a; G 



inf P x limsupZ(X t a ) > = (25) 

which gives, by the positive definitcncss of the function I, the asymptotic stabilizability in the 
large. □ 
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Remark (Uniform asymptotic stabilizability in probability). The existence of a Lyapunov 
function implies a stronger asymptotic stability of the system, which we call uniform asymptotic 
stabilizability. Moreover we will show in a forthcoming paper that the uniform asymptotic stabi- 
lizability can be completely characterized in terms of strict Lyapunov functions. 
The system (CSDE) is uniformly asymptotically stabilizable in probability in O if for every x £ O 
there exists a £ A x such that for every k > 

lim P (sup|X t | > k ) = 0, 
o \t>o J 

$upT«(0\B k ) < +oo, 
xeo 

where T"(C \ B k ) is the expected time spent by the trajectory X in the set 0\B k - 

The fact that the existence of a Lyapunov function implies the uniform asymptotic stabiliz- 
ability follows very easily from the representation formula for the function V and the positive 
definiteness of the function I: 

V(x) > V x f l(X s )ds > T°(0 \ B r ) inf l(y), 
Jo yeO\B r 

which implies 

«.m T*(f) \RU SUp^gQ V(x) NVJIoo 

sup I x {U\B r )< — — - — — . 

xeo mf yG o\B r l\y) mf ae o\B,J(y) 

The proof of the fact that uniform asymptotic stability implies asymptotic stability (in partic- 
ular that for every initial data there exists a control driving asymptotically the trajectory to the 
origin almost surely) is an argument based on continuity properties of trajectories of (CSDE) of 
the type (|2;-S[) we proved in Theorem 1141 



6 Attractors 

Next we extend the results in section 4 to study the stabilizability of general closed sets M C 1R N . 
We denote by d(x, M) the distance between a point x £ M N and the set M. 

We recall that a closed set M is viable with respect to a stochastic controlled dynamical system 
if for every x £ M there exists an admissible control such that the corresponding trajectory remains 
almost surely in M. 

Definition 15 (Stabilizability in probability at M). A closed set M is stabilizable in proba- 
bility for (CSDE) if for every k > there exists 6 > such that, for every x at distance less than 
6 from M , there exists an admissible control a such that the corresponding trajectory X . verifies 

P x ( sup d(X t ,M) > k ) <e. 

\t>0 / 

Remark. We observe that if M is stabilizable in probability according to the previous defini- 
tion, then in particular it is viable. In fact for every e > fixed, the definition gives that, for 
x G M, mf a E x Jq + °° e~~ xt k E (X t )dt = for any A > and any smooth function k e which is non- 
negative, bounded and null on the points at distance less than e from M. By Theorem the 
infimum is attained, therefore for every e > there is a control a £ A x whose corresponding 
trajectory stays almost surely at distance less than s from M: in particular, for every A > 0, 
mf^Es, J*°° e~ u \X t \dt = 0. Therefore, again by Theorem there exists, for every x £ M, a 
minimizing control whose corresponding trajectory stays in M almost surely for all t > 0. 

A geometric characterizations of viability of closed sets with respect to a stochastic differential 
controlled equation has been given in (fj] (see also references therein) . According to this character- 
ization, we note that the fact that the set M is stabilizable in probability implies that the diffusion 
has to degenerate on its boundary: for every x £ dM there exists a £ A such that a(x, a) ■ p = 
for every p generalized normal vector to M at x. 
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We introduce the notion of controlled attractiveness: it coincides, when the system is uncon- 
trolled, with the standard notion of pathwise forward attractiveness (see |21|L 

Definition 16 (Controlled attractor). The set M is a controlled attractor for the system 
(CSDE) in the open set O C M if for every initial data x € O then 



infPz limsupd(X t ",M) > = 0. 

a V t^+oo J 

This means that for every e > there exists a G A x such that the corresponding trajectory 
approaches asymptotically the set M with probability at least 1 — e. 

The set O is called domain of attraction for M : if it coincides with M N the set M is a global 
attractor. 

Remark. We consider a function V : Ml — + JR which satisfies the conditions in the Definition 
1121 of strict global Lyapunov function with the only difference that the function I is assumed only 
nonnegative definite. The proof of Theorem El can be repeated in this case: we obtain that for 
every x € M N 

infP x ( limsupZpf t tt ) > ] = 0. (26) 
We introduce the set C := {y | l(y) = 0}. From (|2l))l we get that for every x S M N 



infPz limsupd(X t Q ,£) > = 0, 

which means that £ is a controlled global attractor for the system. For uncontrolled diffusion 
processes results of this kind can be found in [23] and ^H- The earlier paper of Kushner [2U 
studies also a stochastic version of the La Salle invariance principle, namely, that the omega limit 
set of the process is an invariant subset of C, in a suitable sense. 

We can generalize the notion of control Lyapunov function in order to study the attractiveness 
and the stabilizability of a set M . 

Definition 17 (control M-Lyapunov function). Let M C M N be a closed set and O C M N 
an open set containing M . A function V : O — > [0, +oo) is a control M-Lyapunov function for 
(CSDE) if it satisfies 

(i) it is lower semicontinuous and continuous at every x € dM; 

(ii) it is M-positive definite, i.e., V(x) > for x $ M and V(x) = for x € M; 
(ii) V is M-proper 7 i.e., its level sets {x \ V(x) < fi} are bounded; 

(Hi) it is a viscosity supersolution of the equation 

max {-DV(x) ■ f(x, a) - trace [a(x, a)D 2 V(x)] } > l(x) x G O. 

If 1{ X ) — ^ then V is a control M-Lyapunov function, if l(x) is a M positive definite, Lipschitz 
continuous bounded function then V is a strict control Lyapunov function. 

We can therefore prove for the case of a set M very similar results as for the case of an 
equilibrium point. 

Theorem 18. If the system (CSDE) admits a control M-Lyapunov function V then the system 
is M stabilizable; if moreover the function V is a strict control M-Lyapunov function then the set 
M is a controlled attractor for the system with domain of attraction equal to O . 

Proof. In order to prove this result, one can repeat the proofs given in Theorems 1131 and 1141 since 
the function V satisfies a superoptimality principle and the level sets of V are a local basis of 
neighborhoods of M. □ 
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7 Examples 



In this section we present some very simple examples illustrating the theory. 

The first example is about a stochastic perturbations of stabilizable systems. We apply the 
Lyapunov Theorems to show that an asymptotically controllable deterministic dynamical system 
continues to be stabilizable or asymptotically stabilizable in probability if we perturb it with a 
white noise of intensity small enough. The idea to prove it relies on the fact that, if the stochastic 
perturbation is small enough, then a Lyapunov function for the deterministic systems remains a 
Lyapunov function also for the stochastic one. 

Example 1. We consider a deterministic controlled system in M N 

X t = f(X t ,a t ) (27) 

where f(x,a) is a Lipschitz continuous, locally bounded function in x uniformly with respect to a 
and the control a is a measurable function taking values in a compact space A. We assume that 
the system is globally asymptotically (open loop) stabilizable at the origin, i.e., asymptotically 
controllable in the terminology of deterministic systems |35l Kffij . By the converse Lyapunov 
Theorem |34l 136) . there exists a continuous control Lyapunov function for the system, i.e., for 
some positive definite continuous function L, there exists a proper, positive definite function V 
satisfying in M N 

max {-fix, a) ■ DV} > L(x) (28) 

aeA 

in the viscosity sense. Moreover we can choose the function V to be semiconcave away from the 
origin as proved by Rifford in This means that for every S > there exists a semiconcavity 
constant Cs > such that the function 

V{x)-^\x\* 

is concave in M N \Bs- The semiconcavity constant Cs is an upper bound on the second derivatives 
of the function (to be intended in the sense of distributions). In particular, by the definition of 
semiconcavity, we get that if > 5 and (p, X) G J 2 ~V{x) then 

C s l N - X > 0. (29) 

We study under which conditions the system continues to be asymptotically or Lyapunov 
stabilizable if we perturb it with a white noise. We consider the perturbed system 

dX t = f(X t , a t )dt + a(X t , a t )dB t 

where (B t )t is a Af-dimensional white noise and the function a(x, a) is Lipschitz continuous in x 
uniformly with respect to a and takes values in the space of the N x M dimensional matrices with 
entries in JR. 

By the semiconcavity inequality 1291) and by (|28|l . we get for every \x\ > 8, {p,X) 6 J 2 -~V(x) 
max {—fix, a) ■ p — trace a{x, a)X} > max {—f(x, a) ■ p} — max {trace a(x, a)X} > 

a£A u£A aeA 

> L{x) — Cs max {trace a{x, a)} . 

aeA 

Therefore, if the diffusion a satisfies a small intensity condition 

L(x) 

trace a(x, a) < —p, — Va £ A V \x\ > S, 
Cs 

we can conclude that the function V is a control Lyapunov function for the stochastic system 
and then, according to Theorem 1131 the system is both Lyapunov and Lagrange stabilizable in 
probability. 
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If moreover for every S > 

L(x) 

trace a(x, a) < — - — Va G A V \x\ > S, 

it is possible to construct a positive definite, Lipschitz continuous function / such that V is a 
viscosity supersolution of 

max {—/(a;, a) ■ DV(x) — trace a(x, a)D 2 V(x)} > /(a;) 

and then, by Theorem ll4l the system is asymptotically stabilizable in the large at the equilibrium. 
A similar result can be obtained in the case of local asymptotically controllable systems. 

In the next example we give conditions on a radial function to be a Lyapunov function for the 
stability in probability. 

Example 2. In this example we consider as candidate Lyapunov function for the general con- 
trolled system (CSDE) the function V(x) — |a;| 7 for some positive 7 > and study under which 
conditions the system is stabilizable. 
We compute 

DV(x) = j\x\^~ 2 x D 2 V(x) = 7 |xp" 2 I + 7(7 - 2)\x\~ r ~ i (x ■ x T ). 

Therefore 

max {—f(x, a) ■ DV(x) — trace a{x, a)D 2 V(x)} = 
= j\xp~ 2 max {— f(x, a) ■ x — trace a(x, a) — (7 — 2)\x\~ 2 trace a(x, a)(x ■ ie T ))} = 
= 7|x| 7-2 max {—/(a:, a) ■ x — trace a(x, a) — (7 — 2) | ~ 2 |cr(cc, a) T ■ a;| 2 } . 

If 7 < 2, this gives that the V is a Lyapunov function for the system if for every x there exists 
a *E A such that fix, a) ■ x + trace a(x,a) < 0. We can observe that, since tracea(x,a) > for 
every a, the radial component of the drift / must be everywhere nonpositive, for some a £ A. 
In particular it must be negative to compensate the destabilizing role of the diffusion, whenever 
trace a(x, a) is nonnull. 
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